Skip to content

Implement entity-relation-grondslagen — decision-metadata PATCH endpoint (#1435) - #1503

Merged
rubenvdlinde merged 547 commits into
developmentfrom
feat/1435/entity-relation-grondslagen-impl
Jul 17, 2026
Merged

rubenvdlinde merged 547 commits into
developmentfrom
feat/1435/entity-relation-grondslagen-impl

Conversation

@rjzondervan

Copy link
Copy Markdown
Member

Implements the entity-relation-grondslagen change. Includes a spec rework committed before the implementation — see "Spec rework note" below.

Summary

Introduces an audited, decision-only PATCH endpoint for setting operator decisions on detected-entity occurrences:

  • PATCH /api/entity-relations/{id} — body {bases?: ?array<string>, skipAnonymization?: bool}. Any other top-level key returns HTTP 400 with the offending field name. @NoAdminRequired; the acting user MUST have write-access to the relation's parent file.
  • EntityRelationMapper::updateDecisionMetadata(EntityRelation, array, ?IUser) — single audited write path. Controller passes the loaded relation; mapper enforces the whitelist, validates field shapes, computes a diff, persists changes, and emits exactly one audit-trail entry (only when fields actually change — semantic no-ops produce no audit).
  • Anonymise flow gains skip-awareness. markAsAnonymized adds AND skip_anonymization = 0 to its WHERE; FileTextController::anonymizeFile reads through the new findEntitiesForAnonymization; DocumentProcessingHandler::anonymizeDocument defensively filters skipped values server-side so the contract — "skipped relations are never redacted, full stop" — holds regardless of caller behaviour.

The post-hoc system fields anonymized / anonymizedValue are intentionally NOT in the whitelist; they're set only by markAsAnonymized to preserve the audit invariant "anonymized=true ⟹ the redaction code ran for this row".

Spec rework note

This branch's first commit (ef5464b94) is a substantive amendment to the spec that was approved as PR #1494. The original spec described "the anonymise endpoint accepts bases, persists, then strips before forwarding to OpenAnonymiser". Explore-mode investigation in apply uncovered two architectural mismatches:

  1. OpenAnonymiser is called during detection (EntityRecognitionHandler::detectWithOpenAnonymiser), not at anonymise time. The "strip before forward" requirement had no counterpart in the code.
  2. OR's anonymise endpoint takes a fileId and reads pre-detected relations from the DB; it never forwarded entity arrays anywhere external.

The rework decoupled bases-set from the anonymise call into the new PATCH endpoint, and added the skipAnonymization field on the same audited write path so operators can also opt-out per-occurrence. Both spec and design now describe the actual architecture.

The companion DocuDesk amendment lives on docudesk:feat/123/anonymisation-grondslagen-and-prohibition-gate-amend (local-only — to be pushed alongside this review).

Database

Column Type Default Purpose
bases JSON NULL UUIDs referencing legal grondslagen (consumer-owned vocabulary).
skip_anonymization BOOLEAN false Operator opt-out from the next anonymise pass.

Migration is forward-only and idempotent (hasColumn guards). Existing rows pick up the defaults; no backfill.

Tests

33 unit tests, 133 assertions, all passing against master-nextcloud-1:

  • tests/Unit/Db/EntityRelationTest.php — extended for the two new fields.
  • tests/Unit/Db/EntityRelationMapperTest.php — construction wiring.
  • tests/Unit/Db/EntityRelationMapperUpdateDecisionMetadataTest.php — whitelist, shape validation, semantic-no-op, diff-aware audit emission, audit-failure tolerance, actor resolution (explicit / session / system fallback).
  • tests/Unit/Controller/EntityRelationsControllerTest.php — 401 / 403 / 404 / 400 / 200 / 500 HTTP contract.
$ docker exec -w /var/www/html/apps-extra/openregister master-nextcloud-1 \
    php vendor/bin/phpunit -c phpunit-unit.xml --filter "EntityRelation"
Tests: 33, Assertions: 133 — OK

Quality gates

  • ✓ phpcs clean on all touched files (composer phpcs)
  • ✓ php -l clean
  • ✓ openspec validate entity-relation-grondslagen clean
  • ✓ 33 / 33 unit tests passing
  • ⏸ Live-stack manual smoke deferred to this PR's review phase

Test plan

  • Review the spec-rework commit (ef5464b94) — confirm the decoupled-PATCH architecture matches expectations
  • Review the implementation commit (139e1667d)
  • Apply the migration on a dev install; PATCH a relation with {bases: ["uuid-a"]}, confirm the row is updated and one audit-trail entry exists
  • PATCH a relation with {skipAnonymization: true}; run POST /api/files/:fileId/anonymize on the parent file; confirm the skipped row is NOT redacted and stays anonymized=false
  • PATCH with {anonymized: true}; confirm HTTP 400 with field_not_editable error
  • Confirm pre-change anonymise calls (no skip flags anywhere) behave identically to before
  • Re-review docs/Features/entity-relation-decision-metadata.md for accuracy

Refs: #1435

@github-actions

Copy link
Copy Markdown
Contributor

Quality Report — ConductionNL/openregister @ 554b045

Check PHP Vue Security License Tests
lint ✅
phpcs ✅
phpmd ✅
psalm ✅
phpstan ✅
phpmetrics ✅
eslint ❌
stylelint ✅
composer ✅ ✅ 153/153
npm ✅ ✅ 598/598
PHPUnit ❌
Newman ⏭️
Playwright ⏭️

Quality workflow — 2026-05-12 14:28 UTC

Download the full PDF report from the workflow artifacts.

@WilcoLouwerse WilcoLouwerse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES — Strict mode. 4 blockers (including 1 new from the most recent commit), 5 significant issues.

🔴 Blockers (4): Auth bypass for object/email-bound relations; @NoCSRFRequired on a state-mutating PATCH without justification; ADR-005 violation (getDisplayName() stored as audit actor instead of UID); Newman regression introduced by commit b8949d17 — Newman API Test Suite passed on 139e1667 and now fails with "socket hang up" on every request after the IEventDispatcher constructor parameter was added to EntityRelationMapper without a matching explicit registerService in Application.php.

🟡 Significant (5): Order-sensitive bases comparison causes spurious audit entries; getParams() content-type sensitivity; non-transactional audit write; fileId = 0 bypasses file auth; tasks.md item 3.2a left unchecked despite being implemented in this commit.

🟢 Minor (3): Missing @copyright on 3 new test files (gate-1); missing postSchemaChange() comment in migration; CHANGELOG.md documents the wrong method signature for updateDecisionMetadata.

CI note: PHPUnit failures on quality / PHPUnit (PHP 8.3/8.4, NC stable32) are pre-existing on the development branch. The Newman failure IS new — it regressed in this PR.

Comment thread lib/Db/EntityRelationMapper.php Resolved
Comment thread lib/Db/EntityRelationMapper.php Resolved
@rjzondervan

Copy link
Copy Markdown
Member Author

Re: Newman regression attributed to b8949d17

Wanted to share reproduction results before applying the suggested registerService(EntityRelationMapper::class, …) fix in lib/AppInfo/Application.php, because I cannot reproduce the regression locally and the autowiring path appears healthy.

Tests I ran

  1. Newman crud collection on this branch tip (80c2de5b9):

    • BASE_URL=http://localhost CONTAINER_NAME=master-nextcloud-1 COLLECTIONS="crud" bash tests/newman/run-all.sh
    • Result: 1 collection failed — assertion failures (HTTP 400 on a number of delete operations), no socket hang up, server kept responding.
  2. Same Newman crud collection on development HEAD (9d50f4360 — before my branch diverged):

    • Result: identical set of HTTP 400 failures on the same delete operations.
    • Conclusion: the failures pre-exist b8949d17 and 80c2de5b9. They are not caused by this PR.

Direct evidence the IEventDispatcher autowiring works

Endpoint GET /apps/openregister/api/entities/stats is served by GdprEntitiesController, which constructor-injects EntityRelationMapper (the new signature with IEventDispatcher). Hit it from inside the NC container:

HTTP 200
{...,"byType":{"ORGANIZATION":57,"PERSON":53,"LOCATION":37,"DATE":17,"EMAIL":4,"IBAN":1,"PHONE":1},...}

That request constructs EntityRelationMapper through the DI container with all five deps. If the autowiring chain were broken we'd get a 500 here, not a 200 with a valid payload.

Additionally, sibling mappers (SchemaMapper, MagicMapper, ConfigurationMapper) all use IEventDispatcher via autowiring with no explicit registerService block — same pattern, no failures.

Ask

Could you share the exact failing collection / commit you ran and the failure mode (was it really "socket hang up"? or assertion errors like above?)? I want to make sure I'm not missing something CI-specific before signing off — but I don't want to add a speculative registerService block if the diagnosis is wrong, because that would obscure a different root cause.

Continuing with the other review items (ADR-005 audit actor → UID, PATCH authz, @NoCSRFRequired, the 5 significant items) in parallel.

rjzondervan added a commit that referenced this pull request May 15, 2026
… significant)

Addresses the OR #1503 review on the entity-relation decision-metadata
PATCH endpoint. Four blockers (minus the Newman one — see PR comment),
five significant items, and the minor CHANGELOG/migration cleanups.

**Blockers:**

- **ADR-005 audit actor.** `EntityRelationMapper::emitDecisionMetadataAuditEntry`
  no longer persists `$user->getDisplayName()` on the audit-trail entry's
  `userName` column (PII). Audit entries now record only the UID; the
  display name is dropped.

- **Auth bypass on PATCH /api/entity-relations/{id}.** Object-bound and
  email-bound relations no longer fall through with an accept-with-warning
  log. The controller now constructor-injects `PermissionHandler`,
  `MagicMapper`, `SchemaMapper`, `RegisterMapper`, and `EmailLinkMapper`,
  then routes object-bound relations through
  `PermissionHandler::hasPermission(schema, 'update', uid, owner, object)`
  — the same verdict OR uses everywhere else for object writes — and
  routes email-bound relations through their parent ObjectEntity via the
  EmailLink. Email links without a parent object are denied. `fileId = 0`
  / `objectId = 0` / `emailId = 0` no longer satisfy the file branch.
  Adds `EmailLinkMapper::find(int $id)`.

- **`@NoCSRFRequired`.** Removed from `EntityRelationsController::update`.
  This is a browser-facing decision-time PATCH; Nextcloud's session-CSRF
  protection MUST stay on. Spec amended (D7) to record the decision.

**Significant items:**

- **Order-sensitive `bases` diff.** New private
  `EntityRelationMapper::basesAreEqual(?array, ?array): bool` does
  multiset equality (sort + unique). `['a', 'b']` vs `['b', 'a']` is now
  a semantic no-op and produces no audit entry. `null` vs `[]` remains
  distinct. Storage preserves operator-supplied order; only the diff
  check normalises.

- **`getParams()` content-type sensitivity.** `update()` rejects requests
  whose Content-Type is anything other than `application/json` (or
  empty) with HTTP 415, so the PATCH never depends on body-parser
  heuristics for the decision-time mutation.

- **Non-transactional audit write.** Row UPDATE + audit-trail INSERT
  now run inside a single `$db->beginTransaction()` / `commit()` /
  `rollBack()`. An audit failure rolls back the UPDATE and surfaces as
  HTTP 500 — clients never observe a persisted decision-metadata change
  without a matching audit entry. The downstream event dispatch stays
  OUTSIDE the transaction (post-commit, informational; listener
  failures still don't roll back). Test
  `testAuditEmissionFailureRollsBackUpdateAndThrows` updates the
  contract.

- **`fileId = 0` bypass.** Closed by adding `> 0` guards on `fileId`,
  `objectId`, `emailId` in the authz dispatch helper.

- **`tasks.md` 3.2a + companions checked.** Items 1.1–1.3, 2.1–2.6,
  3.1–3.8 marked `[x]` to reflect what landed in this PR.

**Minor items:**

- **Test copyright headers.** Added SPDX + Conduction headers to
  `EntityRelationsControllerTest`, `EntityRelationMapperTest`,
  `EntityRelationMapperUpdateDecisionMetadataTest`, `EntityRelationTest`
  (also fixed the latter's `namespace Unit\Db` typo).

- **CHANGELOG signature.** Corrected the `updateDecisionMetadata`
  signature in the Unreleased entry from `int $id` to
  `EntityRelation $relation` and noted the transactional contract +
  multiset `bases` diff.

- **Migration `postSchemaChange()` comment.** Documented why
  `Version1Date20260512120000` intentionally does not override
  `postSchemaChange` (nullable column + DB-level default; no backfill).

**Spec.** `design.md` §D6 now records the transactional audit-invariant
+ multiset `bases` equality; §D7 documents the authz wiring (file →
`isUpdateable`; object → `PermissionHandler`; email → parent object via
`EmailLink`) and the `@NoCSRFRequired` removal rationale.

**Tests.** 44 tests / 160 assertions across the four mapper +
controller suites pass. New cases:
- multiset bases equality (reordered, duplicates, null-vs-empty).
- audit-failure rolls back update and rethrows.
- 403 on object-bound when `hasPermission` denies.
- 200 on object-bound when `hasPermission` grants.
- 403 on email-bound when EmailLink has no parent object.
- 403 on a relation with no subject (fileId/objectId/emailId all unset).

**Newman.** The reviewer's diagnosis (commit `b8949d17` broke Newman via
missing `registerService` for autowiring) does not reproduce — see PR
comment. Newman crud fails identically on `development` HEAD (`9d50f4360`,
before this branch existed); the failures are HTTP 400 assertion
errors, not "socket hang up". Autowiring works locally:
`GET /api/entities/stats` (uses `GdprEntitiesController` which
constructor-injects `EntityRelationMapper`) returns HTTP 200 with a
valid payload. Speculative `registerService` block NOT added pending
reviewer's exact failing collection + commit.
@rjzondervan

Copy link
Copy Markdown
Member Author

Review response — commit e34eac991

Addressed every flagged item except the Newman regression, which my earlier comment (#issuecomment-4460023069) showed doesn't reproduce.

Blockers

  • ADR-005 audit actor. EntityRelationMapper::emitDecisionMetadataAuditEntry no longer persists $user->getDisplayName() on the audit-trail entry's userName column. The audit row records only the UID; the display name is dropped (set to null).
  • Auth bypass on PATCH /api/entity-relations/{id}. Object- and email-bound relations no longer fall through with an accept-with-warning log. The controller now constructor-injects PermissionHandler, MagicMapper, SchemaMapper, RegisterMapper, and EmailLinkMapper. Object-bound relations route through PermissionHandler::hasPermission(schema, 'update', uid, owner, object). Email-bound relations resolve their parent ObjectEntity via EmailLink and run the same object-update verdict; email links without a parent object are denied. fileId = 0 / objectId = 0 / emailId = 0 no longer satisfy any branch. Added EmailLinkMapper::find(int $id) to support the email-bound lookup.
  • @NoCSRFRequired. Removed from EntityRelationsController::update. Spec amended (design.md §D7) to record the decision — this is a browser-facing decision-time PATCH and Nextcloud's session-CSRF protection MUST stay on.
  • Newman. Not addressed — see previous comment. The crud Newman collection fails identically on development HEAD (commit 9d50f4360) without any of my commits, with HTTP 400 assertion errors (not "socket hang up"). GET /api/entities/stats (which constructor-injects EntityRelationMapper) returns HTTP 200 locally — autowiring works. I'd push back on this blocker rather than apply a speculative registerService fix that might mask a different CI-environmental cause.

Significant items

  • Order-sensitive bases diff → new private EntityRelationMapper::basesAreEqual(?array, ?array): bool does multiset equality (sort + unique). ['a', 'b'] vs ['b', 'a'] is a semantic no-op and produces no audit entry. null vs [] remains distinct (different operator intent). Storage preserves operator-supplied order; only the diff check normalises.
  • getParams() content-type sensitivity → update() rejects requests with Content-Type other than application/json (or empty) with HTTP 415, so the PATCH never depends on body-parser heuristics for the decision-time mutation.
  • Non-transactional audit write → row UPDATE + audit-trail INSERT now run inside a single $db->beginTransaction() / commit() / rollBack(). An audit failure rolls back the UPDATE and surfaces as HTTP 500 — clients never observe a persisted decision-metadata change without a matching audit entry. The downstream event dispatch stays OUTSIDE the transaction (post-commit, informational; listener failures still don't roll back). Test testAuditEmissionFailureRollsBackUpdateAndThrows updates the contract.
  • fileId = 0 bypass → closed by > 0 guards on fileId / objectId / emailId in the authz dispatch helper.
  • tasks.md 3.2a unchecked → items 1.1–1.3, 2.1–2.6, 3.1–3.8 all marked [x] to reflect what landed in this PR.

Minor items

  • Test copyright headers → SPDX + Conduction headers added to EntityRelationsControllerTest, EntityRelationMapperTest, EntityRelationMapperUpdateDecisionMetadataTest, EntityRelationTest (also fixed the latter's namespace Unit\Db typo).
  • CHANGELOG signature → corrected the updateDecisionMetadata signature from int $id to EntityRelation $relation and noted the transactional contract + multiset bases diff.
  • Migration postSchemaChange() comment → documented why Version1Date20260512120000 intentionally does not override postSchemaChange (nullable column + DB-level default; no backfill needed).

Tests

44 / 44 green, 160 assertions across EntityRelationMapperUpdateDecisionMetadataTest, EntityRelationsControllerTest, EntityRelationTest, EntityRelationMapperTest. New cases cover multiset bases equality, audit-failure rollback, 403 on object-bound denial / 200 on grant, 403 on email-bound without parent object, 403 on relation without any subject.

Comment thread lib/Db/EntityRelationMapper.php Resolved

@WilcoLouwerse WilcoLouwerse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review (Strict). 3 of 4 prior findings resolved: 🔴 @NoCSRFRequired removed and spec amended (design.md §D7); 🔴 auth bypass closed — all three branches (file/object/email) now run an authorisation check, ALL-IDs-null/0 case correctly denies; 🔴 ADR-005 actor — getDisplayName() no longer persisted, audit row records only the UID. Transactional audit-write, HTTP 415 on non-JSON, multiset basesAreEqual, migration idempotency — all verified.

Newman/DI: accepting the pushback that the failure can't be conclusively diagnosed from the diff alone, but downgraded to a 🟡 follow-up because EntityRelationMapper is the only IEventDispatcher-dependent mapper without an explicit registerService() block, inconsistent with the codebase pattern (see inline).

Blocker (1 🔴):

  • Double markAsAnonymized on the HTTP path — commit fb512791 added a markAsAnonymized($fileId, '[REDACTED]') at the end of DocumentProcessingHandler::anonymizeDocument, but FileTextController::anonymizeFile:547 also calls markAsAnonymized with 'anonymized_<date>' afterwards. The second UPDATE overwrites the first's anonymized_value, leaving rows with a value that doesn't match the placeholder actually written to the file.

Concerns (2 🟡):

rubenvdlinde pushed a commit that referenced this pull request May 18, 2026
… significant)

Addresses the OR #1503 review on the entity-relation decision-metadata
PATCH endpoint. Four blockers (minus the Newman one — see PR comment),
five significant items, and the minor CHANGELOG/migration cleanups.

**Blockers:**

- **ADR-005 audit actor.** `EntityRelationMapper::emitDecisionMetadataAuditEntry`
  no longer persists `$user->getDisplayName()` on the audit-trail entry's
  `userName` column (PII). Audit entries now record only the UID; the
  display name is dropped.

- **Auth bypass on PATCH /api/entity-relations/{id}.** Object-bound and
  email-bound relations no longer fall through with an accept-with-warning
  log. The controller now constructor-injects `PermissionHandler`,
  `MagicMapper`, `SchemaMapper`, `RegisterMapper`, and `EmailLinkMapper`,
  then routes object-bound relations through
  `PermissionHandler::hasPermission(schema, 'update', uid, owner, object)`
  — the same verdict OR uses everywhere else for object writes — and
  routes email-bound relations through their parent ObjectEntity via the
  EmailLink. Email links without a parent object are denied. `fileId = 0`
  / `objectId = 0` / `emailId = 0` no longer satisfy the file branch.
  Adds `EmailLinkMapper::find(int $id)`.

- **`@NoCSRFRequired`.** Removed from `EntityRelationsController::update`.
  This is a browser-facing decision-time PATCH; Nextcloud's session-CSRF
  protection MUST stay on. Spec amended (D7) to record the decision.

**Significant items:**

- **Order-sensitive `bases` diff.** New private
  `EntityRelationMapper::basesAreEqual(?array, ?array): bool` does
  multiset equality (sort + unique). `['a', 'b']` vs `['b', 'a']` is now
  a semantic no-op and produces no audit entry. `null` vs `[]` remains
  distinct. Storage preserves operator-supplied order; only the diff
  check normalises.

- **`getParams()` content-type sensitivity.** `update()` rejects requests
  whose Content-Type is anything other than `application/json` (or
  empty) with HTTP 415, so the PATCH never depends on body-parser
  heuristics for the decision-time mutation.

- **Non-transactional audit write.** Row UPDATE + audit-trail INSERT
  now run inside a single `$db->beginTransaction()` / `commit()` /
  `rollBack()`. An audit failure rolls back the UPDATE and surfaces as
  HTTP 500 — clients never observe a persisted decision-metadata change
  without a matching audit entry. The downstream event dispatch stays
  OUTSIDE the transaction (post-commit, informational; listener
  failures still don't roll back). Test
  `testAuditEmissionFailureRollsBackUpdateAndThrows` updates the
  contract.

- **`fileId = 0` bypass.** Closed by adding `> 0` guards on `fileId`,
  `objectId`, `emailId` in the authz dispatch helper.

- **`tasks.md` 3.2a + companions checked.** Items 1.1–1.3, 2.1–2.6,
  3.1–3.8 marked `[x]` to reflect what landed in this PR.

**Minor items:**

- **Test copyright headers.** Added SPDX + Conduction headers to
  `EntityRelationsControllerTest`, `EntityRelationMapperTest`,
  `EntityRelationMapperUpdateDecisionMetadataTest`, `EntityRelationTest`
  (also fixed the latter's `namespace Unit\Db` typo).

- **CHANGELOG signature.** Corrected the `updateDecisionMetadata`
  signature in the Unreleased entry from `int $id` to
  `EntityRelation $relation` and noted the transactional contract +
  multiset `bases` diff.

- **Migration `postSchemaChange()` comment.** Documented why
  `Version1Date20260512120000` intentionally does not override
  `postSchemaChange` (nullable column + DB-level default; no backfill).

**Spec.** `design.md` §D6 now records the transactional audit-invariant
+ multiset `bases` equality; §D7 documents the authz wiring (file →
`isUpdateable`; object → `PermissionHandler`; email → parent object via
`EmailLink`) and the `@NoCSRFRequired` removal rationale.

**Tests.** 44 tests / 160 assertions across the four mapper +
controller suites pass. New cases:
- multiset bases equality (reordered, duplicates, null-vs-empty).
- audit-failure rolls back update and rethrows.
- 403 on object-bound when `hasPermission` denies.
- 200 on object-bound when `hasPermission` grants.
- 403 on email-bound when EmailLink has no parent object.
- 403 on a relation with no subject (fileId/objectId/emailId all unset).

**Newman.** The reviewer's diagnosis (commit `b8949d17` broke Newman via
missing `registerService` for autowiring) does not reproduce — see PR
comment. Newman crud fails identically on `development` HEAD (`9d50f4360`,
before this branch existed); the failures are HTTP 400 assertion
errors, not "socket hang up". Autowiring works locally:
`GET /api/entities/stats` (uses `GdprEntitiesController` which
constructor-injects `EntityRelationMapper`) returns HTTP 200 with a
valid payload. Speculative `registerService` block NOT added pending
reviewer's exact failing collection + commit.
@rubenvdlinde
rubenvdlinde force-pushed the feat/1435/entity-relation-grondslagen-impl branch from 86f2d18 to 0ee978b Compare May 18, 2026 17:35
@rubenvdlinde

Copy link
Copy Markdown
Contributor

Rebased on top of latest development (tip 95aed02c5) to pick up:

Resolved 2 conflicts:

  • openspec/changes/entity-relation-grondslagen/tasks.md: kept PR's full reworked task list (the rework commit on this branch deliberately re-introduces the skip_anonymization column + PATCH endpoint that dev's ADR-032 split removed).
  • CHANGELOG.md (across two rebase steps): merged EML (dev) and entity-relation-grondslagen (PR) entries under ### Added / ### Behaviour changes; on the review-fix commit, kept the newer transaction-wrapped + multiset-equal-diff description.

appinfo/routes.php and lib/Db/EmailLinkMapper.php auto-merged. Force-pushed with lease.

rubenvdlinde pushed a commit that referenced this pull request May 18, 2026
… significant)

Addresses the OR #1503 review on the entity-relation decision-metadata
PATCH endpoint. Four blockers (minus the Newman one — see PR comment),
five significant items, and the minor CHANGELOG/migration cleanups.

**Blockers:**

- **ADR-005 audit actor.** `EntityRelationMapper::emitDecisionMetadataAuditEntry`
  no longer persists `$user->getDisplayName()` on the audit-trail entry's
  `userName` column (PII). Audit entries now record only the UID; the
  display name is dropped.

- **Auth bypass on PATCH /api/entity-relations/{id}.** Object-bound and
  email-bound relations no longer fall through with an accept-with-warning
  log. The controller now constructor-injects `PermissionHandler`,
  `MagicMapper`, `SchemaMapper`, `RegisterMapper`, and `EmailLinkMapper`,
  then routes object-bound relations through
  `PermissionHandler::hasPermission(schema, 'update', uid, owner, object)`
  — the same verdict OR uses everywhere else for object writes — and
  routes email-bound relations through their parent ObjectEntity via the
  EmailLink. Email links without a parent object are denied. `fileId = 0`
  / `objectId = 0` / `emailId = 0` no longer satisfy the file branch.
  Adds `EmailLinkMapper::find(int $id)`.

- **`@NoCSRFRequired`.** Removed from `EntityRelationsController::update`.
  This is a browser-facing decision-time PATCH; Nextcloud's session-CSRF
  protection MUST stay on. Spec amended (D7) to record the decision.

**Significant items:**

- **Order-sensitive `bases` diff.** New private
  `EntityRelationMapper::basesAreEqual(?array, ?array): bool` does
  multiset equality (sort + unique). `['a', 'b']` vs `['b', 'a']` is now
  a semantic no-op and produces no audit entry. `null` vs `[]` remains
  distinct. Storage preserves operator-supplied order; only the diff
  check normalises.

- **`getParams()` content-type sensitivity.** `update()` rejects requests
  whose Content-Type is anything other than `application/json` (or
  empty) with HTTP 415, so the PATCH never depends on body-parser
  heuristics for the decision-time mutation.

- **Non-transactional audit write.** Row UPDATE + audit-trail INSERT
  now run inside a single `$db->beginTransaction()` / `commit()` /
  `rollBack()`. An audit failure rolls back the UPDATE and surfaces as
  HTTP 500 — clients never observe a persisted decision-metadata change
  without a matching audit entry. The downstream event dispatch stays
  OUTSIDE the transaction (post-commit, informational; listener
  failures still don't roll back). Test
  `testAuditEmissionFailureRollsBackUpdateAndThrows` updates the
  contract.

- **`fileId = 0` bypass.** Closed by adding `> 0` guards on `fileId`,
  `objectId`, `emailId` in the authz dispatch helper.

- **`tasks.md` 3.2a + companions checked.** Items 1.1–1.3, 2.1–2.6,
  3.1–3.8 marked `[x]` to reflect what landed in this PR.

**Minor items:**

- **Test copyright headers.** Added SPDX + Conduction headers to
  `EntityRelationsControllerTest`, `EntityRelationMapperTest`,
  `EntityRelationMapperUpdateDecisionMetadataTest`, `EntityRelationTest`
  (also fixed the latter's `namespace Unit\Db` typo).

- **CHANGELOG signature.** Corrected the `updateDecisionMetadata`
  signature in the Unreleased entry from `int $id` to
  `EntityRelation $relation` and noted the transactional contract +
  multiset `bases` diff.

- **Migration `postSchemaChange()` comment.** Documented why
  `Version1Date20260512120000` intentionally does not override
  `postSchemaChange` (nullable column + DB-level default; no backfill).

**Spec.** `design.md` §D6 now records the transactional audit-invariant
+ multiset `bases` equality; §D7 documents the authz wiring (file →
`isUpdateable`; object → `PermissionHandler`; email → parent object via
`EmailLink`) and the `@NoCSRFRequired` removal rationale.

**Tests.** 44 tests / 160 assertions across the four mapper +
controller suites pass. New cases:
- multiset bases equality (reordered, duplicates, null-vs-empty).
- audit-failure rolls back update and rethrows.
- 403 on object-bound when `hasPermission` denies.
- 200 on object-bound when `hasPermission` grants.
- 403 on email-bound when EmailLink has no parent object.
- 403 on a relation with no subject (fileId/objectId/emailId all unset).

**Newman.** The reviewer's diagnosis (commit `b8949d17` broke Newman via
missing `registerService` for autowiring) does not reproduce — see PR
comment. Newman crud fails identically on `development` HEAD (`9d50f4360`,
before this branch existed); the failures are HTTP 400 assertion
errors, not "socket hang up". Autowiring works locally:
`GET /api/entities/stats` (uses `GdprEntitiesController` which
constructor-injects `EntityRelationMapper`) returns HTTP 200 with a
valid payload. Speculative `registerService` block NOT added pending
reviewer's exact failing collection + commit.
@rubenvdlinde
rubenvdlinde force-pushed the feat/1435/entity-relation-grondslagen-impl branch from 0ee978b to 0808d18 Compare May 18, 2026 18:41
rubenvdlinde pushed a commit that referenced this pull request May 18, 2026
… significant)

Addresses the OR #1503 review on the entity-relation decision-metadata
PATCH endpoint. Four blockers (minus the Newman one — see PR comment),
five significant items, and the minor CHANGELOG/migration cleanups.

**Blockers:**

- **ADR-005 audit actor.** `EntityRelationMapper::emitDecisionMetadataAuditEntry`
  no longer persists `$user->getDisplayName()` on the audit-trail entry's
  `userName` column (PII). Audit entries now record only the UID; the
  display name is dropped.

- **Auth bypass on PATCH /api/entity-relations/{id}.** Object-bound and
  email-bound relations no longer fall through with an accept-with-warning
  log. The controller now constructor-injects `PermissionHandler`,
  `MagicMapper`, `SchemaMapper`, `RegisterMapper`, and `EmailLinkMapper`,
  then routes object-bound relations through
  `PermissionHandler::hasPermission(schema, 'update', uid, owner, object)`
  — the same verdict OR uses everywhere else for object writes — and
  routes email-bound relations through their parent ObjectEntity via the
  EmailLink. Email links without a parent object are denied. `fileId = 0`
  / `objectId = 0` / `emailId = 0` no longer satisfy the file branch.
  Adds `EmailLinkMapper::find(int $id)`.

- **`@NoCSRFRequired`.** Removed from `EntityRelationsController::update`.
  This is a browser-facing decision-time PATCH; Nextcloud's session-CSRF
  protection MUST stay on. Spec amended (D7) to record the decision.

**Significant items:**

- **Order-sensitive `bases` diff.** New private
  `EntityRelationMapper::basesAreEqual(?array, ?array): bool` does
  multiset equality (sort + unique). `['a', 'b']` vs `['b', 'a']` is now
  a semantic no-op and produces no audit entry. `null` vs `[]` remains
  distinct. Storage preserves operator-supplied order; only the diff
  check normalises.

- **`getParams()` content-type sensitivity.** `update()` rejects requests
  whose Content-Type is anything other than `application/json` (or
  empty) with HTTP 415, so the PATCH never depends on body-parser
  heuristics for the decision-time mutation.

- **Non-transactional audit write.** Row UPDATE + audit-trail INSERT
  now run inside a single `$db->beginTransaction()` / `commit()` /
  `rollBack()`. An audit failure rolls back the UPDATE and surfaces as
  HTTP 500 — clients never observe a persisted decision-metadata change
  without a matching audit entry. The downstream event dispatch stays
  OUTSIDE the transaction (post-commit, informational; listener
  failures still don't roll back). Test
  `testAuditEmissionFailureRollsBackUpdateAndThrows` updates the
  contract.

- **`fileId = 0` bypass.** Closed by adding `> 0` guards on `fileId`,
  `objectId`, `emailId` in the authz dispatch helper.

- **`tasks.md` 3.2a + companions checked.** Items 1.1–1.3, 2.1–2.6,
  3.1–3.8 marked `[x]` to reflect what landed in this PR.

**Minor items:**

- **Test copyright headers.** Added SPDX + Conduction headers to
  `EntityRelationsControllerTest`, `EntityRelationMapperTest`,
  `EntityRelationMapperUpdateDecisionMetadataTest`, `EntityRelationTest`
  (also fixed the latter's `namespace Unit\Db` typo).

- **CHANGELOG signature.** Corrected the `updateDecisionMetadata`
  signature in the Unreleased entry from `int $id` to
  `EntityRelation $relation` and noted the transactional contract +
  multiset `bases` diff.

- **Migration `postSchemaChange()` comment.** Documented why
  `Version1Date20260512120000` intentionally does not override
  `postSchemaChange` (nullable column + DB-level default; no backfill).

**Spec.** `design.md` §D6 now records the transactional audit-invariant
+ multiset `bases` equality; §D7 documents the authz wiring (file →
`isUpdateable`; object → `PermissionHandler`; email → parent object via
`EmailLink`) and the `@NoCSRFRequired` removal rationale.

**Tests.** 44 tests / 160 assertions across the four mapper +
controller suites pass. New cases:
- multiset bases equality (reordered, duplicates, null-vs-empty).
- audit-failure rolls back update and rethrows.
- 403 on object-bound when `hasPermission` denies.
- 200 on object-bound when `hasPermission` grants.
- 403 on email-bound when EmailLink has no parent object.
- 403 on a relation with no subject (fileId/objectId/emailId all unset).

**Newman.** The reviewer's diagnosis (commit `b8949d17` broke Newman via
missing `registerService` for autowiring) does not reproduce — see PR
comment. Newman crud fails identically on `development` HEAD (`9d50f4360`,
before this branch existed); the failures are HTTP 400 assertion
errors, not "socket hang up". Autowiring works locally:
`GET /api/entities/stats` (uses `GdprEntitiesController` which
constructor-injects `EntityRelationMapper`) returns HTTP 200 with a
valid payload. Speculative `registerService` block NOT added pending
reviewer's exact failing collection + commit.
@rubenvdlinde
rubenvdlinde force-pushed the feat/1435/entity-relation-grondslagen-impl branch from 0808d18 to 6d81c12 Compare May 18, 2026 19:34
Base automatically changed from development to documentation May 18, 2026 20:38
@rjzondervan
rjzondervan changed the base branch from documentation to development May 19, 2026 09:47
rjzondervan added a commit that referenced this pull request May 19, 2026
… + 2 concerns)

🔴 Double markAsAnonymized on the HTTP path — FileTextController::anonymizeFile
called markAsAnonymized($fileId, 'anonymized_<date>') after
DocumentProcessingHandler::anonymizeDocument already called
markAsAnonymized($fileId, '[REDACTED]'). The second UPDATE overwrote the
first, leaving rows with an anonymizedValue that didn't match what was
actually written to the file content. Pick the canonical caller per
option (b) — DocumentProcessingHandler is on every redaction path (HTTP,
DI, event listeners); the controller is bypassed by non-HTTP paths.
Remove the controller-side call; update the two affected FileTextControllerTest
expectations from once() → never() with explanatory comments.

🟡 DI wiring inconsistency — EntityRelationMapper is the only
IEventDispatcher-dependent mapper without an explicit registerService()
block. Other event-dispatcher-dependent mappers (SchemaMapper,
RegisterMapper, MagicMapper, WebhookMapper) are wired explicitly. Add
the EntityRelationMapper block next to MagicMapper, matching the same
pattern. Removes the autowiring question.

🟡 Placeholder format unspec'd + correlation acknowledgement — add two
ADDED Requirements to the spec: (1) the DI anonymise path MUST substitute
each entity using the stable [<TYPE>: <entity_id>] placeholder format,
with three scenarios covering stable id, idempotency on re-anonymise,
and the fallback to UUID fragment for entries missing from the entity
catalogue; (2) DocumentProcessingHandler is the sole canonical caller
of markAsAnonymized in the redaction flow. Also add a Notes section
documenting the cross-document correlation property of the placeholder
choice and the privacy threat model (within-tenant correlation is
in-scope and required by the consumer report).

openspec validate entity-relation-grondslagen — clean.
@rjzondervan

Copy link
Copy Markdown
Member Author

@WilcoLouwerse — addressed in eb02e6b41. Details below mapped to the three findings.

🔴 Double markAsAnonymized on the HTTP path

Picked option (b) per your recommendation: DocumentProcessingHandler::anonymizeDocument is now the sole canonical caller. The controller-side markAsAnonymized($fileId, 'anonymized_<date>') in FileTextController::anonymizeFile is removed, with an inline comment explaining why a controller-side call would race the DI-path call and clobber anonymized_value.

Value semantics: kept '[REDACTED]' (already in DocumentProcessingHandler since fb512791b) as the canonical column marker. The 'anonymized_<date>' timestamp form was incidental — no consumer queries depend on it; the column's purpose is "this file's relations have been redacted at least once" + a generic marker for audit log readability. Per-entity placeholders live in the file content ([<TYPE>: <entity_id>]) and the audit-trail row records the actor + timestamp, so the column doesn't need to carry redundant time info.

Tests updated: FileTextControllerTest::testAnonymizeFileWithSpecialEntities and ::testAnonymizeFileDeduplicatesEntities now assert expects($this->never())->method('markAsAnonymized') on the controller-side mock with explanatory comments. The DocumentProcessingHandler-side call is mocked through $this->fileService in these tests; coverage of the actual mark belongs in DocumentProcessingHandlerTest (which is a separate piece of work; flagged for follow-up if not already there).

🟡 DI wiring — explicit registerService() for EntityRelationMapper

Added the explicit block in Application::registerMappersWithCircularDependencies() immediately after MagicMapper, mirroring the pattern (constructor injects: IDBConnection, AuditTrailMapper, IUserSession, IEventDispatcher, LoggerInterface). Comment explains the rationale (every other event-dispatcher-dependent mapper is explicit; autowiring of framework-interface keys is version-sensitive). Removes the autowiring question entirely.

🟡 [<TYPE>: <entity_id>] placeholder is unspec'd + sequential-DB-integer correlation risk

Two new ADDED Requirements in specs/entity-relation-grondslagen/spec.md:

  1. Placeholder format requirement — captures the exact shape ([<TYPE>: <entity_id>]), the entity_id source (openregister_entities PK, looked up via findEntityIdsByValueForFile), the whitespace contract (exactly one space after the colon), and the fallback to an 8-char UUID fragment when an entity row is missing for DI callers bypassing the standard detection path. Three scenarios: stable placeholder, byte-identical re-anonymise, fallback for missing rows.
  2. Canonical-caller requirement — exactly ONE component (DocumentProcessingHandler::anonymizeDocument) calls markAsAnonymized per redaction. HTTP controllers, DI orchestrators, and event listeners MUST NOT call it independently. Two scenarios covering HTTP-call and DI-call paths.

Plus a Notes section "Placeholder format and cross-document correlation" addressing the correlation concern explicitly:

  • The within-tenant correlation property is deliberate, not a leak — consumer reports (grondslagen-summary in DocuDesk) thread entities through dossier files using these stable ids; the feature wouldn't work without them.
  • Cross-tenant correlation is impossible (per-tenant DB, independent PK sequences).
  • HMAC with per-tenant salt was considered — it would carry the same stability and the same correlation, so doesn't change the privacy property; we kept raw entity_id for simplicity.
  • Per-document salt would defeat correlation but break the consumer use case.

openspec validate entity-relation-grondslagen — clean.

Re-requesting review.

WilcoLouwerse
WilcoLouwerse previously approved these changes May 19, 2026

@WilcoLouwerse WilcoLouwerse left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review verdict — APPROVE

Commit eb02e6b41 addresses the three findings from my 2026-05-18 re-review, and the four findings from the 2026-05-15 round are confirmed resolved by e34eac991. CI Newman API Test Suite is passing on HEAD.

Resolved this round (eb02e6b)

  • 🔴 Double markAsAnonymized on HTTP path — option (b) chosen; DocumentProcessingHandler::anonymizeDocument is sole canonical caller (writes '[REDACTED]'), FileTextController::anonymizeFile no longer marks. Tests now expects($this->never()). Call chain verified end-to-end.
  • 🟡 DI wiring — explicit registerService for EntityRelationMapper — explicit factory block added in Application::registerMappersWithCircularDependencies(), mirroring the pattern used by SchemaMapper / MagicMapper / WebhookMapper.
  • 🟡 Placeholder format unspec'd + correlation risk — two new Requirements in specs/entity-relation-grondslagen/spec.md (placeholder format with three scenarios incl. fallback; sole canonical caller for markAsAnonymized), plus a "Placeholder format and cross-document correlation" Notes section explicitly framing within-tenant correlation as deliberate (Woo dossier consumer requirement), with cross-tenant impossibility and HMAC alternatives discussed.

Resolved earlier (e34eac9)

Minor non-blocking notes for follow-up

  • DocumentProcessingHandler::anonymizeDocument does not emit the warning log that the new spec's fallback scenario says implementations SHOULD log when no matching entity-catalogue row is found. SHOULD-level, non-blocking.
  • The fallback uses $entity['key'] ?? substr(uuid, 0, 8) rather than always a UUID fragment, so a DI caller supplying 'key' can influence the placeholder content. Spec says MAY, so within bounds.

LGTM. 🟢

rjzondervan added a commit that referenced this pull request May 19, 2026
Adds the `react/async` package so the OR combined test branch can resolve
`React\Async\await()` at runtime. The fault path emerged during DocuDesk's
folder-anonymisation flow (entity-relation-grondslagen DI path) where the
async helper is called by ObjectService.

Test-branch-only commit. Not part of any merged PR scope; will land on
development through the appropriate feature branch later (see PR #1503
discussion).
ADR-019 names openregister/scripts/check-integration-parity.sh as the
parity gate, but the canonical check is the JS script shipped in
@conduction/nextcloud-vue. Add a thin bash wrapper that locates and runs
that JS check (via $NEXTCLOUD_VUE_DIR, node_modules, or a sibling
checkout), propagating its exit code. Skips with exit 0 when the JS check
cannot be found locally, since the authoritative gate runs in the
nextcloud-vue repo CI and openregister ships no integration descriptors.
Thijn and others added 27 commits June 3, 2026 15:45
…otes) + .doc/.odt

Rewrite TextExtractionService::extractWord() around a recursive, depth-capped
element walker so table cells (incl. nested tables), in-cell text-runs/list
items, section headers/footers and document-level footnotes/endnotes are all
captured — the previous 2-level traversal silently dropped tables (a PhpWord
Table exposes getRows(), not getElements()).

Select the PhpWord reader from MIME/extension (resolveWordReader): docx->Word2007,
doc->MsDoc, odt->ODText, instead of always using the default Word2007 reader, so
legacy .doc and .odt now load. Route application/vnd.oasis.opendocument.text via
isWordDocument(). Per-document parse failures now log structurally (no PII) and
return null instead of throwing.

Also fixes a pre-existing $emlParser docblock/namespace mismatch flagged by PHPStan.

Adds OpenSpec change text-extraction-word-completeness and unit tests.
…ask checklist

Adds testParseFailureLogsNoDocumentContent (task 6.9) asserting the Word
parse-failure log context is structural-only (file ID / MIME / reader /
exception class) and never leaks document content / extracted text / the
exception message, per ADR-005. Marks the task checklist complete.
…-extraction-word spec

Moves the completed change to openspec/changes/archive/2026-06-03-text-extraction-word-completeness/,
creates the main openspec/specs/text-extraction-word/spec.md from the delta, and
documents Word-family extraction (DOCX/DOC/ODT, full tables/headers/footers/notes)
in docs/features/text-extraction-sources.md.
…tables/headers/footers/notes) + .doc/.odt support' (#54) from feat/text-extraction-word-completeness into development

automerge: openregister#54
… feature/backend-accept-ISO-8601 into development

automerge: openregister#53
…Result + IUser mock compat)' (#52) from fix/iresult-nc34-test-mock into development

automerge: openregister#52
#1997)' (#51) from feat/1997/pdf-anonymisation into development

automerge: openregister#51
…48) from feat/1342/validate-self-folder-access into development

automerge: openregister#48
…roactive hydra-build)' (#46) from hydra/proactive-system-context-owner-attribution into development

automerge: openregister#46
…+ mydash->launchpad refs' (#6) from rename/openbuilt-and-mydash-refs into development

automerge: openregister#6
Rule 0c covering test for the global integration-registry bootstrap
listener introduced by the universal-shared-integration-registry spec.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…l bootstrap (#34)' (#55) from feature/34/universal-shared-integration-registry into development

automerge: openregister#55
…ovider + SSE stream + Message.context (#12)' (#62) from feature/12/ai-chat-companion-orchestrator into development
…om DOCX/ODT during anonymisation

Adds an OfficeDocumentSanitizer that runs ahead of the entity-anonymisation
walker for .docx / .odt inputs. It performs XML-level surgery on a temp copy
of the ZIP container (the original Nextcloud file is never mutated) to remove
the PII-bearing non-text structures the walker is structurally blind to.

Strips / scrubs:
- comments (word/comments*.xml + people.xml + inline markers; ODT office:annotation)
- tracked changes — accept inserts (keep text), drop deletions (remove text),
  strip w:rsid* revision attributes (ODT text:change-start/end + text:change)
- custom XML parts (customXml/item*.xml) + unwrap data-bound content controls
- document metadata (docProps core/app/custom.xml, ODT meta.xml) -> sentinel
  "DocuDesk Anonymisation" (design D5; defends against Word's metadata re-fill)
- person field codes AUTHOR/USERNAME/USERINITIALS/LASTSAVEDBY (simple + complex
  OOXML forms; ODT author-name/initials/initial-creator)
- hyperlinks flattened to plain text, URL + relationship dropped (design D7)

[Content_Types].xml and _rels/*.rels are reconciled in a single pass per index
part (ZipArchive does not surface pending in-session writes to a later
getFromName), so the output opens cleanly in Word/LibreOffice. ODT is now an
anonymisation-supported format (previously corrupted by the raw-string path).

A SanitizationReport (PII-free per-category counts, JsonSerializable) is
retained on DocumentProcessingHandler::getLastSanitizationReport(); logging is
counts-only (ADR-005); XML parsing is XXE-safe; encrypted documents raise a
typed SanitizationException surfaced as a caller-correctable error.

Architecture notes:
- Strategy pattern (SanitizerInterface / DocxSanitizer / OdtSanitizer) +
  OfficeDocumentSanitizer orchestrator, autowired into DocumentProcessingHandler.
- No anonymisation-log table exists in the current codebase, so the spec's
  AnonymizationLog JSON-column persistence (tasks §5/§10) is deferred and the
  report is exposed via the handler getter instead (documented in tasks.md).
- Fixtures are synthesised in-test (PdfMetadataSanitizerTest convention).
- Manual Word/LibreOffice reopen gate (§11) + live-NC integration (§10) deferred
  to a reviewer with desktop Office readers / the Docker test env.

composer check: phpcs/phpmd/psalm clean on all new files; phpstan clean on new
files (pre-existing ddn\sapp\PDFDoc resolution errors are environmental — SAPP
absent from the build vendor). 22 new unit tests pass (5 orchestrator tests
skip cleanly without the NC server classes). openspec validate clean.
…ty-relation-grondslagen-impl

# Conflicts:
#	CHANGELOG.md
#	lib/Service/File/DocumentProcessingHandler.php
…structures from DOCX/ODT (hydra-build)' (#63) from hydra/office-document-sanitization into development
…ixes-for-pipelinq into development

Reviewed-on: https://codeberg.org/Conduction/openregister/pulls/64
Reviewed-by: rjzondervan <rjzondervan@noreply.codeberg.org>
…ty-relation-grondslagen-impl

# Conflicts:
#	lib/Service/File/DocumentProcessingHandler.php
…' (#49) from feat/1435/entity-relation-grondslagen-impl into development

Reviewed-on: https://codeberg.org/Conduction/openregister/pulls/49
Reviewed-by: WilcoLouwerse <wilcolouwerse@noreply.codeberg.org>
@rubenvdlinde
rubenvdlinde merged commit 0ce24ed into development Jul 17, 2026
@rubenvdlinde
rubenvdlinde deleted the feat/1435/entity-relation-grondslagen-impl branch July 17, 2026 09:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants